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SUBJECT:  Final  Report  on  the  Audit  of  Management  of  the  Software 

Technology  for  Adaptable,  Reliable  Systems  Program 
(Report  No.  91-050  ) 


This  is  our  final  report  on  the  Audit  of  Management  of  the 
Software  Technology  for  Adaptable,  Reliable  Systems  Program 
(STARS,  or  the  Program)  for  your  information  and  use.  Comments 
on  a  draft  of  this  report  were  considered  in  preparing  the  final 
report.  The  primary  audit  objective  was  to  determine  whether 
management  of  STARS  was  effective  and  efficient.  Secondary 
objectives  were  to  assess  the  Program's  progress  and  evaluate  the 
adequacy  of  related  internal  controls.  The  audit  was  made  from 
October  1989  through  June  1990.  STARS  is  one  of  three  major  DoD 
efforts  to  reduce  the  costs  of  Mission-Critical  Computer 
Resources  (MCCR)  software.  The  Program  was  begun  in  1983; 
through  1990,  management  spent  about  $119  million  on  research  and 
development  to  improve  software  productivity,  quality,  and 
reliability.  In  1990,  DoD  spent  an  estimated  $30  billion  on  MCCR 
software,  and  these  costs  may  reach  $42  billion  by  1995. 

Overall,  we  concluded  that  increased  managerial  attention  is 
needed  for  STARS  to  meet  its  goals  in  a  timely  manner.  The 
results  of  the  audit  are  summarized  in  the  following  paragraphs, 
and  the  details,  audit  recommendations,  and  management  comments 
are  in  Part  II  of  this  report. 

While  the  audit  showed  that  the  management  of  STARS  has 
recently  improved,  the  Program  had  not  progressed  as  planned. 
Following  a  major  restructuring  in  1986,  STARS  was  to  have  been 
completed  by  1992.  However,  because  of  substantial  delays, 
management  plans  to  continue  STARS  through  1995.  STARS  was 
affected  by  funding  shortages.  Although  STARS  was  chartered  as  a 
joint  DoD  program.  Service  participation  was  minimal,  and 
coordination  with  related  research  efforts  needed  to  be  enhanced 
to  ensure  efficient  use  of  resources.  Additionally,  STARS  had 
not  developed  clear  goals  and  firm  indicators  of  the  Program's 
progress  or  value,  and  internal  controls  and  oversight  of  the 
Program  were  not  effective.  STARS  management  had  begun 
addressing  these  issues,  but  a  more  unified  approach  by  DoD  was 
needed.  Since  a  DoD  Software  Master  Plan  is  being  developed,  we 


recommended  that  the  Under  Secretary  of  Defense  for  Acquisition 
direct,  and  provide  appropriate  support  for,  the  Defense  Advanced 
Research  Projects  Agency  (DARPA)  to  identify  adequate  resources 
for  timely  Program  completion.  We  recommended  that  the  Under 
Secretary  direct  more  Service  participation,  provide  for  STARS 
evaluations,  and  place  related  technical  activities  of  the  DoD 
Software  Initiative  under  single  management.  We  also  recommended 
that  DARPA  establish  a  system  to  measure  STARS  progress  and 
improve  related  internal  controls  (page  5). 

We  issued  a  draft  of  this  report  on  August  31,  1990,  and 
requested  comments  from  the  Under  Secretary  of  Defense  for 
Acquisition  and  the  Director,  DARPA.  On  behalf  of  the  Under 
Secretary  of  Defense  for  Acquisition,  the  Director  of  Defense 
Research  and  Engineering  provided  comments  on  November  14,  1990, 
and  we  received  DARPA' s  comments  on  November  20,  1990.  Both 
responses  were  in  compliance  with  DoD  Directive  7650.3.  The 
Comptroller  of  the  Department  of  Defense  also  made  comments, 
although  we  did  not  request  them.  All  comments  are  shown  in 
Appendixes  B,  C,  and  D. 

The  Director  of  Defense  Research  and  Engineering  generally 
commented  that  the  draft  report  was  inaccurate  and  misleading 
because  it  focused  on  problems  of  the  past  and  did  not  fairly 
present  recent  improvements  in  the  management  of  STARS. 
Additionally,  the  Director  felt  that  our  citation  of  a  draft  DoD 
Software  Master  Plan  was  inappropriate,  and  that  some  recommen¬ 
dations  exceeded  the  audit's  scope.  Specifically,  the  Director 
of  Defense  Research  and  Engineering  nonconcurred  with 
Recommendation  l.d.,  which  addressed  consolidated  management  of 
the  technical  activities  of  the  Ada  Joint  Program  Office,  the 
Software  Engineering  Institute,  and  STARS.  He  stated  that  the 
activities  of  each  are  not  necessarily  closely  related,  and  that 
they  are  adequately  coordinated.  The  Director  concurred  with  the 
intent  of  Recommendation  l.c.,  which  addressed  enhanced  oversight 
of  STARS  by  DoD,  but  believed  that  recent  DARPA  changes  in 
management  structure  and  personnel  provided  adequate  reviews  to 
annually  determine  the  Program's  continuation.  The  Director 
fully  or  partially  concurred  with  Recommendations  l.a.(l)., 
l.a.(2).,  and  l.b.,  which  focused  on  more  direct  Service 
involvement  in  funding  and  managing  the  Program.  The  Director 
cited  resource  constraints  and  uncertain  program  execution  roles 
for  the  Services,  but  he  provided  responsive  action  plans  and 
completion  dates  that  met  the  recommendations'  intent. 

We  continue  to  believe  that  Recommendations  l.c.  and  l.d. 
are  sound.  Our  intent  is  to  ensure  effective  DoD  oversight  of  a 
major  software  research  program  and  the  more  efficient  use  of 
limited  resources.  However,  based  on  the  comments  received  and 
on  subsequent  discussions  with  management,  we  revised  our 
recommendations  to  express  this  intent  more  clearly. 
Accordingly,  we  request  that  the  Under  Secretary  of  Defense  for 
Acquisition  comment  on  our  revised  Recommendations  l.c.  and  l.d. 
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The  Director,  DARPA  concurred  with  our  recommendations 
except  for  Recommendation  2.a.(l).,  which  required  STARS  to 
establish  quantitative  goals.  DARPA  stated  that  quantitative 
goals  would  be  artificial  and  proposed  that  clear  goals  for  each 
of  STARS'  technical  areas  would  be  more  meaningful.  We  believe 
the  proposal  is  a  workable  alternative  to  achieving  our  intent, 
and  have  revised  Recommendation  2.a.(l).  in  the  final  report 
accordingly.  We  request  that  DARPA  comment  on  this  revised 
recommendation. 

Although  not  a  draft  report  addressee,  the  Comptroller  of 
the  Department  of  Defense  concurred  with  all  recommendations. 

The  audit  identified  internal  control  weaknesses  as  defined 
by  Public  Law  97-255,  Office  of  Management  and  Budget  Circular 
A-123,  and  DoD  Directive  5010.38.  DARPA  concurred  with  our 
related  recommendations  and  provided  responsive  corrective 
actions  and  estimated  completion  dates.  These  actions,  when 
completed,  should  correct  the  internal  control  deficiencies 
identified . 

DoD  Directive  7650.3  requires  that  all  audit  recommendations 
be  resolved  promptly.  Accordingly,  final  comments  on  the 
unresolved  issues  in  this  report  should  be  provided  within 
60  days  of  the  date  of  this  memorandum.  Your  comments  should 
indicate  concurrence  or  nonconcurrence  with  the  finding  and  each 
unresolved  recommendation.  If  you  concur,  describe  the 
corrective  actions  already  taken  or  planned  and  give  the 
completion  dates  for  actions  already  taken  and  estimated  dates 
for  completion  of  planned  actions.  If  you  nonconcur,  state  your 
specific  reasons.  If  appropriate,  you  may  propose  alternative 
methods  for  accomplishing  desired  improvements.  The  audit 
quantifies  no  potential  monetary  benefits,  but  Appendix  E 
summarizes  nonmonetary  benefits. 
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The  audit  team  (listed  in  Appendix  G)  appreciates  the 
courtesies  extended.  If  you  have  any  questions  about  this  audit, 
please  contact  Mr.  Terry  McKinney  at  (703)  693-0430 
(AUTOVON  223-0430)  or  Mr.  James  Hutchinson  at  (703)  693-0452 

(AUTOVON  223-0452).  Copies  of  the  this  report  are  being  provided 
to  the  activities  listed  in  Appendix  H  and  will  be  made  available 
to  other  interested  parties. 

Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


cc : 

Secretary  of 
Secretary  of 
Secretary  of 


the  Army 
the  Navy 
the  Air  Force 
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MANAGEMENT  OF  THE  SOFTWARE  TECHNOLOGY 
FOR  ADAPTABLE,  RELIABLE  SYSTEMS  PROGRAM 

PART  I  -  INTRODUCTION 


Background 

Mission-Critical  Computer  Resources  (MCCR)  are  computer  hardware 
and  software  components  that  are  integral  to  the  functioning  of 
weapon  systems  and  other  critical  military  and  intelligence 
systems.  Almost  all  DoD  weapon  and  support  systems  depend  on 
MCCR  subsystems  for  some  degree  of  operational  control.  Software 
(the  instructions  and  data  definitions  that  enable  computer 
hardware  to  function)  has  become  an  important  and  costly  part  of 
DoD' s  mission-critical  systems.  Between  1990  and  1995,  MCCR 
hardware  is  expected  to  cost  about  $6  billion  annually,  while 
related  software  costs  are  expected  to  grow  from  about 
$30  billion  to  $42  billion.  Further,  because  software  has 
considerable  control  over  system  functions,  software  deficiencies 
often  affect  the  performance  and  delivery  of  weapon  systems. 
Problems  with  performance  and  delivery  can  cost  much  more  than 
the  software  itself.  In  the  next  few  years,  economic  pressures 
and  personnel  shortages  will  probably  result  in  more  automation 
of  DoD's  mission-critical  systems. 

The  role  of  software  in  contemporary  weapon  systems  is 
substantial  and  complex  and  is  increasing  in  importance.  While 
the  C5-A  aircraft  required  about  25,000  lines  of  software,  about 
750,000  lines  of  software  are  needed  for  the  C-17A,  DoD's  newest 
airlifter.  Planners  expect  that  the  Advanced  Tactical  Fighter 
will  require  nearly  7  million  lines  of  software.  Advances  in 
technology  have  provided  most  of  the  required  hardware 
capabilities,  but  have  not  kept  pace  with  DoD's  escalating  needs 
for  affordable  and  reliable  MCCR  software.  In  1987*  the  Defense 
Science  Board  Task  Force  on  Military  Software  reported  that: 

...The  "smarts"  of  smart  weapons  are  provided  by  soft¬ 
ware.  Software  is  crucial  to  intelligence, 

communications,  command,  and  control...  The  chief 
"military  software  problem"  is  that  we  cannot  get 
enough  of  it,  soon  enough,  reliable  enough,  and  cheap 
enough  to  meet  the  demands  of  weapon  systems  designers 
and  users.... 

DoD's  software  problem  has  caused  cost  overruns  and  has  delayed 
the  fielding  of  weapon  systems.  In  the  late  1970 's  and  early 
1980 's,  DoD  began  major  initiatives  to  alleviate  MCCR  software 
problems.  One  initiative  was  the  Software  Technology  for 
Adaptable,  Reliable  Systems  Program  (STARS,  or  the  Program), 
which  was  established  to  improve  the  productivity,  quality,  and 
adaptability  of  MCCR  software.  The  Program  focused  on  improving 
the  technology  and  processes  that  support  the  creation  and 


evolution  of  software.  STARS  is  a  research  and  development 
program  with  three  primary  thrusts:  to  adapt  modern  processes  for 
developing  and  maintaining  software;  to  provide  highly  automated 
environments  for  creating  and  maintaining  software;  and  to 
efficiently  reuse  previously  developed  software. 

STARS  was  funded  in  1983  and  chartered  in  1984  as  a  joint  DoD 
program.  After  the  Program  was  substantially  revised  in  1986, 
management  planned  for  STARS  to  end  in  1992;  STARS'  results  were 
to  be  transferred  to  the  Services'  functional  organizations 
beginning  in  1989.  A  total  budget  of  about  $188  million  was 
expected  for  FY's  1986  through  1990.  By  January  1990,  however, 
STARS  management  expected  that  the  Program  would  continue  through 
FY  1995.  For  the  period  FY  1986  through  FY  1990,  STARS  will  have 
cost  about  $99  million,  and  if  planned  work  is  executed,  DoD  will 
have  invested  over  $160  million  in  STARS  by  the  end  of  FY  1995. 

Objectives  and  Scope 

Our  primary  audit  objective  was  to  determine  if  management  of  the 
STARS  Program  was  effective  and  efficient.  Secondary  audit 
objectives  were  to  assess  STARS'  progress  in  achieving  its  goals 
and  to  evaluate  the  adequacy  of  related  internal  controls.  We 
reviewed  STARS  records  and  related  records  created  since  STARS' 
inception,  focusing  on  the  period  August  1986  through  June  1990. 

To  evaluate  program  management  and  assess  STARS'  progress,  we 
examined  plans,  schedules,  reports,  and  other  program 
documentation.  We  discussed  managerial,  administrative,  and 
technical  aspects  of  STARS  with  the  Office  of  the  Secretary  of 
Defense,  the  Defense  Advanced  Research  Projects  Agency  (DARPA), 
the  Services,  and  other  DoD  and  Federal  officials.  We  also 
discussed  STARS  with  contractors  and  other  knowledgeable 
private-sector  personnel.  We  did  not  perform  detailed  audit  work 
at  other  DoD  activities  where  similar  research  was  being  done  on 
software . 

We  examined  and  analyzed  DARPA' s  internal  control  procedures  and 
requirements  related  to  program  oversight  and  review.  We  also 
identified  and  evaluated  STARS  program  management  controls.  We 
reviewed  and  analyzed  STARS  management  control  procedures  and 
requirements  for  program  administration,  contract  administration, 
and  program  review.  Appendix  F  lists  the  activities  we  visited 
or  contacted. 

This  program  audit  was  made  from  October  1989  through  June 
1990.  The  audit  was  made  in  accordance  with  auditing  standards 
issued  by  the  Comptroller  General  of  the  United  States  as 
implemented  by  the  Inspector  General,  DoD,  and  accordingly 
included  such  tests  of  the  internal  controls  as  were  considered 
necessary . 
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Internal  Controls 


We  reviewed  DARPA's  internal  controls  for  the  oversight  and 
management  of  the  STARS  Program.  We  found  that  DARPA's  overall 
program  management  control  procedures  and  requirements  were 
generally  adequate ,  but  had  not  been  applied  to  STARS  and  needed 
to  be  strengthened.  Although  STARS  qualified  as  a  major  program, 
DARPA  had  not  designated  it  as  such;  therefore,  STARS  management 
was  not  required  to  develop  detailed  program  plans  and  undergo 
frequent  review  by  DARPA  officials.  Additionally,  DARPA  had  not 
developed  procedures  to  ensure  that  major  programs  are 
consistently  identified. 

We  also  evaluated  STARS'  program  management  controls.  Program 
management  controls  had  not  been  updated  since  1986,  and 
associated  procedures  and  requirements  did  not  reflect  the 
substantial  changes  in  STARS  since  that  time.  Additionally,  the 
objectives  of  the  control  procedures  were  not  uniformly  met. 
Program  documentation  was  seriously  deficient.  Management  could 
not  readily  provide  historical  fiscal  records,  show  how  project 
expenditures  met  STARS  goals  and  objectives,  or  show  the  need  for 
substantial  changes  that  had  been  made  in  the  Program.  Also,  the 
Program's  relationship  with  other  organizations  had  not  been 
established  through  memorandums  of  understanding. 

Completion  of  actions  begun  by  management  during  the  audit  and 
implementation  of  our  recommendations  in  Part  II  of  this  report 
should  correct  the  internal  control  deficiencies. 

Prior  Audit  Coverage 

We  identified  no  prior  audits  of  STARS,  but  the  Program  has  been 
the  subject  of  past  reviews  and  studies.  "An  Assessment  of  the 
STARS  Program,  September-October  1985,"  issued  in  December  1985 
by  the  Institute  for  Defense  Analysis,  concluded  that  STARS  faced 
critical  managerial  problems  and  that  the  Program  should 
encourage  more  industry  involvement  in  achieving  its  objectives. 
The  study  recommended  that  a  program  director  be  appointed  and 
given  appropriate  authority.  It  also  recommended  that  STARS 
shift  its  focus  away  from  developing  complete,  wide-spectrum 
environments,  and  toward  the  development  of  a  general  framework 
for  using  a  variety  of  software  tools  and  the  production  of 
compatible  sets  of  tools.  These  recommendations  led  to  a 
restructuring  of  STARS,  and  were  largely  reflected  in  the 
Program's  management  and  technical  plans,  which  were  developed  in 
1986. 
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In  1984,  the  Defense  Science  Board  Task  Force  on  Military 
Software  was  asked  to  address  managerial  and  technical  changes 
needed  to  improve  the  software  acquisition  process  within  DoD, 
and  was  also  asked  to  assess  STARS.  The  Task  Force  concluded 
that  STARS  was  an  important  effort,  but  needed  to  be  better 
coordinated  with  other  DoD  initiatives  in  software  technology  and 
methodology;  that  more  specific  goals  and  plans  needed  to  be 
developed;  and  that  Program  management  needed  to  be  strengthened. 

The  Task  Force  recommended; 

-  the  creation  of  a  Joint  Program  Office  to  oversee  STARS, 
the  Ada  Joint  Program  Office,  and  the  Software  Engineering 
Institute; 

-  that  the  major  software  technology  programs  develop  a 
coordinated  plan; 

that  STARS  management  define  a  new  set  of  program  goals 
and  develop  an  implementation  plan  which  emphasized  visible, 
early  milestones;  and 

that  STARS  management  choose  several  software  programs  in 
early  development  phases  and  augment  the  funding  of  the  programs 
to  ensure  the  use  of  modern  software  practices  and  tools. 

As  discussed  in  Part  II  of  our  report,  these  recommendations  have 
not  been  fully  implemented. 
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PART  II  -  FINDING  AND  RECOMMENDATIONS 


Program  Management 


FINDING 


Management  of  the  Software  Technology  for  Adaptable,  Reliable 
Systems  Program  (STARS,  or  the  Program)  was  not  fully  effective 
and  efficient  in  planning,  developing,  executing,  and  controlling 
STARS.  Resource  constraints  and  inadequate  coordination  hindered 
program  implementation  as  set  forth  in  the  most  recent  STARS 
Program  Management  Plan.  Also,  while  the  Program  was  chartered 
to  improve  software  productivity  and  quality,  management  had  not 
established  a  performance  baseline  and  quantitative  methods  of 
measuring  improvements.  Inadequate  management  controls  also 
contributed  to  program  difficulties.  Achievement  of  the 
Program's  goals  and  realization  of  its  potential  benefits  may  not 
be  possible  without  increased  management  attention. 

DISCUSSION  OF  DETAILS 


Background.  STARS  was  funded  in  1983  and  was  chartered  in 
1984  as  a  joint  DoD  program.  After  making  little  progress  during 
its  first  few  years,  the  Program  underwent  several  reviews.  In 
1986,  the  Program's  fourth  manager  was  appointed  and  given 
enhanced  execution  authority.  STARS'  technical  thrusts  were 
revised  and  the  changes  were  incorporated  into  the  Program's 
technology  and  management  plans'.  STARS  was  to  develop  software 
technology  and  demonstrate  program  results  through  four  major 
activities . 


-  Different  technology  approaches  would  be  developed  by 
competing  prime  contractors,  and  management  would  choose  the  best 
alternative  for  the  contractors  to  develop  cooperatively. 

-  Funds  would  be  provided  to  the  Services  for  software 
research  and  development  consistent  with  STARS  goals. 

-  Through  its  Foundations  projects  (contracts  to  develop 
several  software  tool  prototypes),  the  Program  would  enhance  the 
existing  base  of  tools  for  Ada-based  systems. 

-  To  demonstrate  and  measure  the  benefits  of  these 
efforts,  management  planned  a  series  of  Shadow  projects  (parallel 
development  of  mission-critical  applications,  using 
STARS-developed  software  engineering  products  and  processes)  to 
be  executed  by  the  Services. 

In  April  1988,  the  STARS  Joint  Program  Office  was  transferred 
from  the  Office  of  the  Secretary  of  Defense  (OSD)  to  the  Defense 
Advanced  Research  Projects  Agency  (DARPA).  Also  in  1988,  the 
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Program's  prime  contractors  were  competitively  selected.  The 
overall  goal  of  STARS  is  to  increase  the  productivity,  quality, 
and  reliability  of  MCCR  software  by  adapting  modern  software 
engineering  practices  to  the  specification,  development, 
operation,  maintenance,  and  retirement  of  DoD  software.  To 
achieve  this  goal,  management  has  directed  the  contractors  to 
focus  on  three  primary  areas:  improved  software  engineering 
processes  and  associated  technologies;  adaptable  software 
engineering  environments  using  integrated,  commercially-available 
tools;  and  improved  repositories  and  processes  for  software 
reuse . 

The  DoD  Software  Initiative  commonly  refers  to  STARS  and 
two  other  programs.  The  three  programs  are  interrelated  and  are 
aimed  at  reducing  the  costs  of  MCCR  software.  The  Ada  Joint 
Program  Office  (AJPO) ,  managed  by  the  Director  of  Defense 
Research  and  Engineering,  maintains  and  updates  the  Ada 
programming  language.  DoD  developed  Ada  specifically  to  reduce 
the  life-cycle  costs  of  software;  in  1983,  Ada  was  designated  as 
DoD's  standard  language  for  MCCR  applications.  In  1984,  DoD 
established  the  Software  Engineering  Institute  (SEI)  at 
Carnegie-Mellon  University.  Its  purposes  are  to  develop  software 
engineering  concepts  with  high  potential  and  to  promote  the  use 
of  advanced  software  engineering  techniques  and  methods 
throughout  the  MCCR  software  industry.  Like  STARS,  the  SEI  is 
funded  and  managed  by  DARPA. 

Resource  Constraints.  The  execution  of  STARS,  as  planned  in 
1986,  was  slowed  by  factors  beyond  the  program  director's 
control.  Planned  financial  resources  did  not  materialize.  For 
FY's  1986  through  1990,  management  planned  on  resources  of  about 
$188  million;  however,  only  $105  million  was  allocated.  Further, 
about  $6  million  of  this  allocation  was  reprogrammed  to  other 
projects,  leaving  about  $99  million  available  for  STARS. 
Appendix  A  illustrates  the  planned  and  actual  funds  available  for 
each  major  Program  activity.  These  shortages  have  reduced 
research  by  the  prime  contractors,  eliminated  complementary 
software  research  efforts  by  the  Services,  and  ended  the  Shadow 
projects . 

Program  management  contracted  with  three  prime  contractors  to 
competitively  develop  alternate  technology  approaches  to  solving 
problems.  However,  lack  of  funding  has  reduced  the  number  of 
approaches  that  contractors  can  competitively  explore. 

Another  objective  of  management  was  to  enable  software  to  be 
securely  developed  using  distributed  processing  at  physically 
separate  locations,  with  different  hardware  used  at  each 
location.  However,  funds  have  not  been  available  to  accomplish 
this  objective.  Security  requirements  should  be  addressed  during 
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the  initial  development  phase.  When  security  measures  have  to  be 
developed  and  retrofitted  at  a  later  date,  the  costs  can  be  50  to 
100  times  greater. 

To  encourage  Service  involvement  and  participation,  the 

1986  STARS  Program  Management  Plan  stated: 

...Funding  has  been  allocated  to  each  Service  to  fund 
STARS  efforts  and  participation  by  Service 
laboratories .. .to  help  achieve  the  cost  and  quality 
benefits  sought  through  STARS  in  Service  programs.... 

For  FY's  1986  through  1990,  the  amount  of  funding  planned  for 
Shadow  projects  was  about  $24  million;  about  $38  million  was 
planned  for  other  STARS-related  research  efforts.  Through 
FY  1988,  management  allocated  about  $7  million  to  the  Services 
for  six  Shadow  projects  and  about  $14  million  for  related 
software  research  projects.  In  FY  1989,  management  stopped 
allocations  to  the  Services  and  has  not  restored  funding, 
although  the  Services  have  made  several  requests  for  funds  to 
complete  research  projects. 

In  our  opinion,  funding  limitations  will  continue  to  affect  the 
STARS  Program.  STARS  management  has  made  plans  to  produce  three 
fully  functional  software  engineering  environments,  each 
configured  to  support  a  specific  functional  software  domain  such 
as  command  and  control  or  avionics.  Projected  funding  is  not 
adequate  to  achieve  this  objective.  A  software  engineering 
environment  being  developed  by  the  Army  has  cost  about 
$30  million,  and  substantial  further  investment  will  be  required 
to  complete  this  effort.  Additionally,  a  complete  environment 
for  developing  large  administrative  software  systems  is  estimated 
to  cost  about  $50  million.  For  FY's  1991  through  1994,  STARS' 
total  budget  is  about  $61  million. 

The  STARS  Joint  Program  Office  (JPO)  was  not  staffed  according  to 
the  1986  plans.  These  plans  showed  that  the  STARS  JPO  would 
consist  of  a  director,  deputy  directors  from  each  of  the 
Services,  and  technical  and  clerical  staff.  Appropriate 
personnel  positions  were  allocated,  but  the  positions  were  never 
fully  staffed.  The  Navy  assigned  a  deputy  director  on  a 
part-time  basis.  The  Defense  Logistics  Agency  also  provided 
administrative  assistance.  When  the  Program  was  transferred  from 
OSD  to  DARPA,  this  limited  support  was  lost,  and  STARS  was 
managed  by  one  person.  In  our  opinion,  one  person  cannot 
effectively  manage  a  program  such  as  STARS. 

Related  Programs.  Although  STARS  was  chartered  as  a  joint 
program,  there  has  been  little  active  participation  by  the 
Services.  Also,  STARS  activities  have  not  been  well-coordinated 
with  other  software  research.  This  has  led  to  the  funding  of 
similar  (possibly  duplicate)  research  projects  within  DoD. 
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Additionally,  AJPO,  SEI,  and  STARS  managers  need  to  better 
coordinate  their  efforts,  since  their  program  goals  are  closely 
related. 

As  previously  discussed,  the  Services  provided  little  support  to 
the  STARS  JPO.  Service  program  managers  administered  STARS- 
sponsored  Shadow  projects  and  other  research,  but  the  STARS  JPO 
was  severely  understaffed,  and  the  Program  was  a  joint  DoD  effort 
in  name  only.  As  a  result,  STARS  contributed  funds  to  Service 
projects,  but  these  projects  were  not  effectively  coordinated. 
Present  management  no  longer  considers  STARS  a  joint  program  and 
intends  to  alter  the  Program's  charter  accordingly.  However, 
STARS  management  has  recently  begun  better  coordination  with 
other  Service  research  projects,  such  as  software  repositories 
and  methods  of  measuring  software  attributes. 

The  preliminary  draft  of  the  DoD  Software  Master  Plan,  issued 
February  9,  1990,  outlines  a  consolidated  DoD  approach  and 
actions  needed  to  address  problems  resulting  from  the  escalating 
use  and  costs  of  software  in  defense  systems.  Part  of  the  Plan 
attempts  to  identify  and  categorize  all  of  DoD's  unclassified 
software  research  efforts.  It  summarizes  research  projects  on 
several  aspects  of  software,  from  the  shortage  of  qualified 
software  personnel  to  software  security.  The  plan  lists  several 
projects  that  are  similar  to  STARS  activities  and  may  duplicate 
them.  For  instance,  one  of  the  main  thrusts  of  STARS  is  the 
development  of  software  engineering  environments.  At  least 
17  other  environment-related  projects  are  listed  in  the  Software 
Master  Plan. 

Since  STARS  and  SEI  goals  are  similar,  STARS  management  planned 
in  1986  to  work  closely  with  SEI  to  coordinate  technical 
developments  and  transition  the  Program's  products  into  early 
use.  The  relationship  was  initially  successful,  but  declined  due 
to  misunderstandings  and  funding  difficulties.  In  March  1989, 
the  SEI  terminated  STARS-sponsored  work,  which  included 
preliminary  efforts  to  measure  and  compare  the  effectiveness  of 
Shadow  projects.  STARS  management  has  begun  efforts  to  involve 
SEI  in  STARS  activities  again.  In  our  opinion,  SEI  can 
contribute  to  the  software  engineering  processes  being  developed 
by  STARS.  SEI  participation  can  also  speed  up  the  widespread  use 
of  STARS  products  by  the  MCCR  software  community. 

In  1987,  to  strengthen  the  AJPO,  SEI,  and  STARS,  and  to  permit 
easier  coordination  of  common  goals  and  objectives,  the  Defense 
Science  Board's  Task  Force  on  Military  Software  recommended  that 
DoD  place  the  three  programs  under  common  management.  DoD 
officials  did  not  implement  the  recommendation  because  AJPO's 
primary  mission  was  policy-making  rather  than  technical,  and 
moving  the  AJPO  might  convey  the  impression  that  DoD  was  not 
committed  to  using  Ada.  We  believe  these  factors  are  no  longer 
fully  valid.  We  found  little  reason  to  doubt  DoD's  commitment  to 
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Ada.  Additionally,  while  most  of  AJPO's  technical  activities  are 
directly  related  to  maintaining  the  Ada  programming  language  and 
standard,  some  are  not.  By  bringing  the  related  technical 
activities  of  the  DoD  Software  Initiative  under  a  single  manager, 
these  technical  efforts  could  be  better  coordinated  and  more 
efficiently  executed.  For  instance,  we  believe  that  the  AJPO’s 
Ada  Technology  Insertion  Project  could  be  used  to  demonstrate  the 
usefulness  of  STARS  products  as  well  as  the  suitability  of  the 
Ada  programming  language  in  meeting  the  requirements  of  MCCR 
applications . 

As  previously  noted,  management  has  recently  focused  on  improved 
coordination  with  the  Services  and  with  related  research 
programs.  However,  we  believe  that  a  more  integrated  DoD 
approach  to  STARS  and  other  software  technology  programs  would 
result  in  more  efficient  use  of  DoD  resources,  and  would  help  to 
ensure  that  some  issues  not  directly  addressed  by  STARS  receive 
attention  in  other  research. 

Program  Goals  and  Measurements.  STARS  management  had  not 
specified  goals,  established  a  system  for  measuring  progress  in 
goal  achievement,  or  effectively  demonstrated  its 
accomplishments.  Management  did  not  plan  to  quantify  or  fully 
demonstrate  program  results  until  STARS  nears  completion.  In  the 
interim,  evaluating  STARS'  progress  is  highly  subjective. 

STARS'  goals  were  to  improve  the  productivity,  quality,  and 
reliability  of  MCCR  software;  to  promote  the  development  and 
application  of  reusable  software;  and  to  reduce  the  costs  and 
development  time  of  defense  software.  However,  management  had 
specified  only  one  goal,  a  tenfold  increase  in  productivity. 
Management  also  had  not  established  a  performance  baseline  from 
which  progress  could  be  quantified  or  evaluated. 

One  of  the  Program's  original  thrusts  was  to  measure  improvements 
resulting  from  the  processes  and  products  produced  by  the 
Program.  However,  STARS  had  not  selected  and  implemented  a 
system  to  measure  software  attributes,  primarily  because  the 
software  industry  has  no  measurement  standards.  In  the  absence 
of  concrete  standards,  numerous  methods  of  software  measurement 
have  been  developed  and  are  selectively  used  by  many  software 
managers.  STARS  management  needed  to  select,  implement,  and 
continually  use  a  measurement  system  to  establish  a  performance 
baseline  and  track  goal  accomplishment  and  program  progress. 
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In  1985,  the  STARS  Goals  and  Objectives  Working  Group  reported 
that  STARS: 

...has  the  responsibility  for  demonstrating 

improvement  over  the  current  situation.  It  should 
establish  exactly  what  the  current  baseline  is  and 
keep  this  knowledge  up  to  date  as  improvements  are 
made  over  time.  It  should  assure  demonstrations 
showing  improvement  (as  opposed  to  demonstrations 

showing  value)  are  made.  In  particular,  it  should 
assure  that  improvement  in  the  productivity, 

reusability,  reliability  and  adaptability  are 

demonstrated .... 

A  series  of  12  Shadow  projects,  starting  in  1987,  was  to 
demonstrate  the  gains  resulting  from  STARS-produced  software 
tools  and  environments.  Because  of  contracting  delays,  however, 
enhanced  software  tools  could  not  be  developed  quickly  enough  to 
be  used  in  the  Shadow  projects.  Although  the  Shadow  projects 
have  developed  Ada-based  software  for  weapon  systems,  they  have 
not  demonstrated  gains  in  software  productivity,  reliability,  or 
adaptability  resulting  from  the  use  of  STARS  products. 

Because  there  was  little  evidence  of  concrete  achievement,  the 
Program  was  increasingly  criticized  inside  and  outside  DoD. 
Management  needed  to  more  effectively  demonstrate  interim 
accomplishments.  A  measurement  system  would  help  track  program 
progress,  provide  a  better  base  for  program  execution,  and 
support  funding  requests.  Additionally,  STARS  economic  value 
cannot  be  determined  until  its  improvements  are  quantified. 

Internal  and  Program  Management  Controls.  Internal  controls 
over  and  within  STARS  needed  to  be  strengthened.  To  assure  that 
the  Program  is  executed  in  a  timely  manner,  senior  DARPA 
management  needed  to  review  STARS  more  closely  and  require  more 
detailed  planning  information  from  STARS  management. 
Additionally,  STARS'  program  management  controls  were  outdated 
and  did  not  ensure  that  control  objectives  were  met. 

DARPA  Instruction  13,  "DARPA  Program  Management  Procedures," 
gives  program  management  procedures  and  requirements.  It 
requires  DARPA-sponsored  research  programs  that  exceed  life-cycle 
costs  of  $100  million,  or  are  of  other  special  interest,  to  be 
designated  as  major  programs.  Major  programs  must  develop  more 
detailed  program  plans  and  undergo  more  frequent  review  by  senior 
■DARPA  officials.  STARS  had  not  been  designated  as  a  major 
program  by  DARPA;  therefore,  STARS  program  reviews  were  not  as 
frequent  or  as  stringent  as  those  required  for  a  major  program. 
Reviews  focused  on  budgetary  issues,  not  on  program  execution  or 
progress.  Records  were  available  for  only  the  most  recent  review 
of  STARS;  records  of  prior  reviews  had  been  destroyed  because  of 
limited  file  space. 
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We  were  informed  that  the  Director  of  DARPA  determined  whether  a 
program  should  be  classified  as  major.  However,  we  could  not 
identify  procedures  for  presenting  the  Director  with  information 
about  criteria  for  major  programs.  To  ensure  that  DARPA  programs 
received  appropriate  oversight,  procedures  needed  to  be  developed 
and  implemented  that  would  help  the  Director  make  consistent 
decisions  on  program  classification. 

The  1986  STARS  Program  Management  Plan  described  program 
management  control  objectives,  procedures,  and  requirements. 
Although  one  of  the  requirements  was  that  the  plan  be  updated 
annually,  management  had  not  updated  it.  As  a  result,  the  plan 
and  its  management  controls  were  outdated.  It  did  not  reflect 
the  organizational,  financial,  and  program  execution  changes  in 
STARS  since  1986.  Additionally,  procedures  for  maintaining 
adequate  program  documentation  were  ineffective. 

Management  could  not  provide  adequate  records  of  the  Program's 
fiscal  history.  Financial  summaries  and  records  prior  to  1986 
were  unavailable,  and  although  the  records  for  1986  through  April 
1988  showed  STARS  budgets  and  allocations,  they  contained  little 
information  about  expenditures.  Since  STARS  reallocated  most  of 
its  funds  to  the  Services  during  this  period,  we  asked  the 
Services  to  explain  how  they  spent  the  funds.  Only  the  Air  Force 
could  readily  account  for  its  use  of  STARS  funds.  Although  we 
found  no  evidence  of  fraud,  we  could  not  identify  the  ultimate 
use  of  about  $4.3  million  in  STARS  funds.  Additionally,  we  could 
not  tell  whether  STARS  management  had  required  that  funds 
allocated  to  the  Services  be  spent  to  further  STARS  goals  or  meet 
any  specific,  agreed-upon  needs. 

Since  STARS  had  been  transferred  to  DARPA,  DARPA' s  financial 
accounting  system  and  internal  procedures  provided  an  audit  trail 
of  expenditures.  However,  STARS  program  files  did  not  give  the 
program  director  (under  DARPA,  the  Program  Manager)  the 
information  he  needed  about  STARS'  financial  and  program 
execution  history  to  avoid  repeating  past  mistakes. 

Program  Improvements.  During  our  audit,  program  officials 
independently  initiated  several  actions  to  improve  STARS 
management  and  responded  to  some  of  our  concerns.  These  actions 
should  improve  program  effectiveness  and  increase  the  Program's 
chances  of  success. 

STARS'  contracting  agent,  the  Air  Force's  Electronic  Systems 
Division,  has  improved  its  administration  of  STARS'  prime 
contracts.  Reviews  of  contractor  actions  and  deliverables  have 
been  formalized  and  strengthened.  Of  the  209  deliverables  due  on 
or  before  January  11,  1990,  38  (18  percent)  were  more  than 
90  days  past  due.  On  April  30,  1990,  29  (10  percent)  of 
299  deliverables  were  more  than  90  days  past  due.  Actions  have 
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been  initiated  to  further  improve  contractors'  timeliness. 
Increased  technical  expertise  is  available  to  monitor 
contractors,  and  subcontracting  procedures  have  been  established 
to  reduce  or  prevent  delays. 

DARPA  placed  STARS  in  its  Information  Science  and  Technology 
Office,  where  STARS  and  other  related  research  can  be  better 
coordinated.  Working  relationships  with  the  Services  and  other 
related  organizations  have  been  improved,  and  memorandums  of 
understanding  are  being  negotiated  to  formalize  the 
responsibilities  and  interests  of  each.  Management  has  arranged 
for  technical  and  program  advice  from  the  Joint  Advisory 
Committee,  which  is  composed  of  senior  representatives  from  each 
Service.  To  promote  the  visibility  of  STARS,  several  prototypes 
of  software  tools  developed  under  STARS  have  been  released  to  the 
public.  Further,  management  has  begun  updating  the  Program 
Management  Plan. 

Conclusions.  Although  DoD  has  generally  supported  the  aims 
of  STARS,  management  has  not  been  able  to  execute  the  Program  as 
planned.  Many  difficulties,  such  as  short  management  tenures  and 
declining  resources,  have  been  beyond  the  control  of  management 
and  have  hindered  the  Program's  progress.  STARS  addresses  broad 
and  complex  issues  that  will  not  be  resolved  overnight.  However, 
the  Program  has  produced  little  firm  evidence  of  interim 
accomplishment,  and  the  need  for  its  continuation  has  been 
questioned. 

Recent  management  initiatives  should  help  correct  some  of  STARS' 
problems.  However,  considering  the  risks  of  a  research  program 
such  as  STARS,  its  history  of  problems,  and  the  program  and 
technical  issues  still  unresolved,  these  actions  should  be 
speeded  up.  Additionally,  enhanced  program  oversight  and  review 
would  help  ensure  that  STARS  accomplishes  its  goals  in  an 
effective  and  timely  manner. 

Improving  the  cost-effectiveness  and  quality  of  DoD  software  is  a 
worthwhile  goal,  but  an  integrated  DoD  decision  on  the  future 
role  of  STARS,  and  an  appropriate  level  and  course  of  action,  are 
needed.  To  make  the  best  use  of  limited  resources,  DoD  needs  to 
respond  to  a  basic  question:  Is  the  next  investment  in  STARS 
worthwhile,  or  can  the  funds  be  better  applied  elsewhere? 

RECOMMENDATIONS  FOR  CORRECTIVE  ACTION 


1.  We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition,  in  consonance  with  the  DoD  Master  Software  Plan 
under  development: 

a.  Direct,  and  provide  appropriate  support  for,  the  Defense 
Advanced  Research  Projects  Agency  to: 
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(1)  identify,  in  coordination  with  the  Services, 
adequate  resources  for  the  Software  Technology  for  Adaptable, 
Reliable  Systems  Program,  and 

(2)  develop  concrete  financial  plans  to  complete  the 
Software  Technology  for  Adaptable,  Reliable  Systems  Program  in  a 
timely  manner. 

b.  Direct  each  of  the  Services  to  assign  a  representative 
to  work  directly  and  be  collocated  with  the  Program  Manager  of 
the  Software  Technology  for  Adaptable,  Reliable  Systems  Program. 

c.  Provide  for  an  independent  annual  evaluation  of  the 
Software  Technology  for  Adaptable,  Reliable  Systems  Program.  The 
evaluation  should  focus  on  the  Program's  progress  in  meeting  its 
objectives  and  goals,  and  whether  its  accomplishments  merit 
Program  continuation. 

d.  Consolidate  the  management  of  related  technical 
activities  that  could  more  efficiently  serve  the  joint  purposes 
of  the  Ada  Joint  Program  Office,  the  Software  Engineering 
Institute,  and  the  Software  Technology  for  Adaptable,  Reliable 
Systems  Program. 

2.  We  recommend  that  the  Director,  Defense  Advanced  Research 
Projects  Agency: 

a.  Require  the  Software  Technology  for  Adaptable,  Reliable 
Systems  Program  to: 

(1)  establish  clear  goals  for  each  technical  area, 

(2)  document  a  performance  baseline,  and 

(3)  select  and  implement  a  system  for  measuring 
progress  in  goal  achievement. 

b.  Require  the  Program  Manager  to  develop,  document,  and 
maintain  an  effective  system  of  management  control  objectives, 
requirements,  and  procedures  unique  to  the  Software  Technology 
for  Adaptable,  Reliable  Systems  Program  and  incorporate  them  into 
a  revised  Program  Management  Plan. 

c.  Designate  the  Software  Technology  for  Adaptable, 
Reliable  Systems  Program  as  a  major  program  and  manage  it 
accordingly. 

d.  Develop  and  implement  procedures  to  consistently 
identify  major  programs  as  defined  in  Defense  Advanced  Research 
Projects  Agency  Instruction  No.  13. 
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MANAGEMENT  COMMENTS 


The  Director  of  Defense  Research  and  Engineering  (the  Director) 
generally  commented  that  the  report  was  inaccurate,  unfairly 
criticized  program  execution,  focused  on  past  problems  and  did 
not  adequately  recognize  program  management  improvements 
initiated  during  the  audit  period,  repeated  a  prior 
recommendation,  and  was  not  limited  to  its  stated  scope.  The 
Director  also  stated  that  it  was  inappropriate  to  make  our 
recommendations  in  terms  of  the  draft  DoD  Software  Master  Plan 
since  it  had  not  been  finalized  and  approved. 

In  responding  to  our  specific  recommendations,  the  Director  fully 
or  partially  concurred  with  Recommendations  l.a.(l).,  l.a.(2)., 
and  l.b.  and  provided  action  plans  that  met  the  recommendations' 
intent  and  should  be  completed  by  the  end  of  FY  1991.  While 
citing  budget  restrictions,  the  Director  stated  that  additional 
funding  has  been  identified  in  DARPA's  budget  submission;  when 
combined  with  investments  in  compatible  technology  by  STARS 
contractors,  this  funding  is  adequate  to  make  STARS  viable. 
STARS  management  is  developing  a  detailed  execution  plan,  which 
will  be  reviewed  to  determine  if  funding  requests  are  reasonable. 
The  plan  will  provide  more  linkage  between  products  and  funding 
required.  Additionally,  STARS  management  and  the  Services  are 
defining  the  Services'  involvement  in  program  execution. 
Commensurate  Service  financial  and  program  management  roles  will 
be  determined  later. 

The  Director  concurred  with  the  intent  of  draft  report 
Recommendation  l.c.,  which  required  an  annual  program  evaluation 
by  high-level  DoD  officials,  but  believes  that  an  adequate  review 
framework  currently  exists  due  to  recent  DARPA  changes  in 
management  structure  and  personnel.  The  Director  nonconcur  red 
with  Recommendation  l.d.  in  the  draft  report,  which  addressed 
consolidated  management  of  the  DoD  Software  Initiative's 
technical  activities.  The  Director  felt  that  the  AJPO's 
technical  activities  were  not  closely  related  to  those  of  the  SEI 
or  STARS  and  stated  that  there  is  adequate  coordination  with  the 
STARS  Program.  The  complete  text  of  comments  from  the  Director 
of  Defense  Research  and  Engineering  is  shown  at  Appendix  B. 

The  Director,  Defense  Advanced  Research  Projects  Agency  concurred 
with  Recommendations  2. a. (2).,  2. a. (3).,  2.b.,  2.c.,  and  2.d. 
DARPA  provided  action  plans  for  each  recommendation,  with  all 
actions  to  be  completed  by  the  end  of  FY  1991.  DARPA  is 
formulating  plans  to  develop  a  performance  baseline  and  has 
tasked  the  SEI  to  develop  the  framework  for  measuring  Program 
progress  and  achievements.  An  internal  control  system  for  the 
STARS  Program  is  being  developed  and  will  be  set  forth  in  the 
revised  Program  Management  Plan.  The  revised  STARS  Program 
Management  Plan  will  also  serve  as  the  detailed  program  plan 
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required  by  DARPA  when  it  designates  STARS  as  a  major  program. 
DARPA  will,  as  part  of  ongoing  review  of  its  management  systems, 
develop  and  implement  procedures  to  consistently  identify  major 
programs.  DARPA  partially  concurred  with  Recommendation 
2 .a . ( 1 ) . ,  which  required  STARS  to  establish  quantitative  program 
goals.  Stating  that  quantified  goals  would  be  artificial,  DARPA 
proposed  that  establishing  clear  goals  for  each  technical  area 
would  be  more  meaningful.  The  complete  text  of  comments  provided 
by  the  Defense  Advanced  Research  Projects  Agency  is  shown  at 
Appendix  C. 

The  Comptroller  of  the  Department  of  Defense  concurred  with  all 
draft  report  recommendations,  and  the  complete  text  of  the 
comments  is  shown  at  Appendix  D. 

AUDIT  RESPONSE  TO  MANAGEMENT  COMMENTS 

After  our  draft  report  was  issued,  we  met  with  management 
officials  to  better  define  the  inaccuracies  cited  in  the  response 
from  the  Director  of  Defense  Research  and  Engineering. 
One  example  was  the  number  of  STARS  program  directors  between 
1983  and  1986.  The  Director  of  Defense  Research  and  Engineering 
believed  that  our  source  of  information  was  incorrect  and  that 
there  were  three  different  program  directors  before  the  major 
program  reorganization  in  1986,  not  the  four  implied  in  our  draft 
report.  We  believe  that  either  number  indicates  a  troubled 
program.  The  Director  also  believed  that  our  description  of  the 
objective  of  an  improved  software  engineering  process  was  too 
narrow.  We  agree  that  our  description  reflected  a  prior  STARS 
Program  focus.  For  both  points,  we  have  appropriately  revised 
the  final  report. 

However,  we  continue  to  believe  that  the  audit  report  fairly  and 
objectively  presents  STARS'  problems  and  related  improvements. 
We  agree  with  the  Director  of  Defense  Research  and  Engineering 
that  the  Program's  management  was  strengthened  during  our  audit, 
but  many  recent  management  initiatives  are  not  complete. 
Therefore,  the  results  of  these  initiatives  and  attendant 
reductions  in  program  risk  are  not  conclusive.  We  also  believe 
the  report  clearly  presents  the  impact  of  resource  shortages  on 
planned  STARS  execution. 

We  cited  the  draft  DoD  Software  Master  Plan  because  it  represents 
DoD's  first  consolidated  approach  to  addressing  the  extremely 
high  costs  of  software.  We  doubt  that  the  final  Master  Plan  will 
differ  substantially  from  the  draft.  Therefore,  we  believe  the 
draft  DoD  Software  Master  Plan  can  and  should  be  used  to  focus 
and  guide  the  high-level  administration  of  DoD  software  research. 

Our  report  recognizes  recent  managerial  improvements  initiated  by 
DARPA  and  STARS.  Additionally,  the  implementation  of  the 
recommendations  that  DARPA  has  concurred  with  should  further 
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strengthen  the  management  of  STARS.  We  continue  to  believe/ 
however,  that  past  program  difficulties,  the  risks  involved,  and 
the  relative  lack  of  clearly  demonstrated  accomplishment  show  a 
need  for  more  oversight  of  the  STARS  Program.  STARS  holds  the 
promise  of  substantial  contribution  to  the  goals  of  the  draft  DoD 
Software  Master  Plan,  and  while  the  Program's  past  and  potential 
costs  are  high,  STARS  has  not  been  subjected  to  intensive 
review.  In  our  opinion,  these  factors  merit  thorough,  high-level 
DoD  review  of  the  Program  on  a  periodic  basis.  We  also  believe 
that  increased  high-level  involvement  with  STARS  would  provide 
more  informed  and  coordinated  DoD  execution  of  the  finalized  DoD 
Software  Master  Plan. 

Our  report  does  not  duplicate  an  earlier  recommendation  by  the 
Defense  Science  Board.  The  Defense  Science  Board  recommended  the 
joint  management  of  all  activities  of  the  AJPO,  SEI,  and  STARS; 
we  recommend  that  the  manager  responsible  for  STARS  and  SEI  also 
be  in  charge  of  the  AJPO's  technical  projects  that  are  not 
directly  related  to  Ada  maintenance.  While  the  AJPO  was  not 
specifically  included  in  our  announced  audit  objectives,  we 
believe  that  a  recommendation  directed  at  the  AJPO  is 
appropriate,  since  the  AJPO  is  a  component  of  the  DoD  Software 
Initiative.  We  also  continue  to  believe  that  related  technical 
activities  of  the  AJPO  and  STARS  can  be  more  efficiently  managed 
on  a  consolidated  basis.  Contrary  to  the  Director's  assertion, 
we  found  very  little  indication  that  the  activities  of  the  AJPO 
and  STARS  were  coordinated.  While  the  AJPO's  focus  is  on  the 
maintenance,  improvement,  and  standardization  of  the  Ada 
programming  language,  it  administers  projects  that  we  believe  are 
closely  related  to  STARS  activities.  For  instance,  the  AJPO  has 
a  current  project  concerning  the  Common  Ada  Programming  Support 
Environment  Interface  Standard,  which  is  also  central  to  STARS' 
efforts  to  establish  a  framework  for  interfacing  the  various 
components  of  a  software  engineering  environment.  As  discussed 
in  the  report,  we  also  believe  that  the  AJPO's  Ada  Technology 
Insertion  Project  could  be  used  for  the  joint  purposes  of  the  two 
programs. 

The  intent  of  our  draft  report  recommendations  was  to  provide 
effective  DoD  oversight  of  a  major  software  research  program  and 
to  ensure  the  more  efficient  use  of  limited  DoD  resources.  Based 
on  management  comments,  we  have  revised  the  disputed  recommen¬ 
dations  in  the  final  report  to  better  express  our  intent. 
Accordingly,  we  request  that  the  Under  Secretary  of  Defense  for 
Acquisition  reconsider  his  position  on  these  issues  and  respond 
to  our  revised  Recommendations  l.c.  and  l.d.  in  the  final  report. 

DARPA's  response  included  an  acceptable  alternative  for 
implementing  the  intent  of  Recommendation  2.a.(l).  Accordingly, 
we  have  revised  this  recommendation  in  the  final  report  to 
reflect  the  planned  actions,  and  we  ask  that  DARPA  respond  to  the 
revised  recommendation. 
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*  Actual  includes  $4.3  million  for  which 
we  could  not  identify  an  ultimate  use. 


This  page  was  left  out  of  original  document 


DIRECTOR  OF  DEFENSE  RESEARCH  AND  ENGINEERING 
WASHINGTON,  DC  20301-3010 


1  4  NOV  1990 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  Report  on  the  Audit  of  the  STARS  Program 

(Project  No.  OFE-0007) ,  dated  August  31,  1990. 


We  recognize  the  significant  effort  expended  to  obtain  an 
understanding  of  the  background  and  evolution  of  the  Software 
Technology  for  Adaptable,  Reliable  Systems  (STARS)  Program. 
However,  some  of  the  subject  draft  audit  report  findings  are 
inaccurate,  and  some  of  its  recommendations  are  inappropriate. 
During  the  report  period,  October  1989  to  June  1990,  efforts  had 
already  been  made  to  correct  many  of  the  past  problems  cited  in 
the  report.  While  there  is  some  acknowledgement  in  the  report 
that  management  of  the  program  has  improved,  its  recommendations 
appear  to  be  influenced  by  problems  of  the  past.  We  believe  the 
STARS  Program  has  made  significant  progress  since  the  time  of  the 
audit  in  the  areas  of  program  management  structure,  the  program 
management  team  and  the  existing  controls  to  successfully  manage 
and  complete  the  current  program. 

Although  we  commonly  refer  to  the  software  development 
process,  it  is  not  a  singular  well-defined  set  of  methods  and 
activities,  and  is  not  yet  a  true  engineering  discipline. 

However,  our  dependence  on  software  is  such  that  we  must  act  to 
improve  the  process  whether  it  embodies  a  mature  engineering 
discipline  or  not.  Under  these  circumstances,  we  should  expect 
that  a  research  and  development  program  like  STARS,  whose  main 
objective  is  to  develop  means  to  improve  this  process,  will  be 
refined  and  adjusted  as  our  experience  with  the  process  and  its 
evolving  methods,  techniques  and  tools  grows. 

Remedies  to  most  of  the  problems  cited  in  the  Audit  Report 
have  already  been  implemented  by  the  current  Program  Manager. 

Our  specific  responses  to  the  recommendations  are  provided  in  the 
attachment . 


(iuoJb 

Charles  M.  Herzfeld 

Attachment 
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COMMENTS  ON 

OIG  DRAFT  REPORT  ON  THE  AUDIT  OF  THE  STARS  PROGRAM 
(PROJECT  NO.  OFE-0007) ,  DATED  AUGUST  31,  1990 

GENERAL  COMMENTS 


The  Audit  Report  on  the  Software  Technology  for  Adaptable, 
Reliable  Systems  (STARS)  Program  (Project  No.  OFE-0007)  expended 
significant  effort  to  obtain  an  understanding  of  the  background 
and  evolution  of  the  program.  However,  some  of  its  findings  are 
inaccurate,  and  some  of  its  recommendations  are  inappropriate. 
During  the  report  period,  October  1989  to  June  1990,  efforts  had 
already  been  made  to  correct  many  of  the  past  problems  cited  in 
the  report.  While  there  is  some  acknowledgement  in  the  report 
that  management  of  the  program  has  improved,  its  recommendations 
appear  to  be  influenced  by  problems  of  the  past.  We  believe  the 
STARS  Program  has  made  significant  progress  since  the  time  of  the 
audit  in  the  areas  of  program  management  structure,  the  program 
management  team  and  the  existing  controls  to  successfully  manage 
and  complete  the  current  program. 

In  one  instance,  the  report  has  merely  repeated  an  earlier 
recommendation  to  consolidate  the  Ada  Program,  the  STARS  Program, 
and  the  Software  Engineering  Institute  Program  without  additional 
rationale.  This  recommendation  was  first  proposed  by  a  Defense 
Science  Board  Task  Force  under  different  circumstances.  At  that 
time,  the  recommendation  was  specifically  evaluated  in  light  of 
the  Ada  Joint  Program  Office’s  mission,  and  rejected  by  OSD 
management  as  inappropriate.  An  explanation  of  the  rationale  for 
rejection  is  included  in  the  response  to  the  recommendations 
below.  Furthermore,  this  recommendation  was  outside  the  intended 
scope  of  the  Audit  Report. 

The  report  cites  a  major  restructuring  of  the  program  in 
1986,  and  mentions  the  fact  that  continued  shortfalls  in  program 
funding  have  caused  continued  cutbacks  and  adjustments  in  the 
planned  effort  and  objectives.  The  criticism  that  STARS 
management  did  not  execute  the  program  as  originally  planned  was 
inappropriate,  since  replanning  has  necessarily  occurred  along 
the  way.  A  valid  criticism  would  be  that  STARS  Program 
Management  should  have  formally  documented  revisions  of  the 
program  objectives  and  plans  each  year  as  they  occurred.  The 
current  Program  Manager  is  in  the  process  of  formally  updating 
the  Program  Plan.  It  will  be  available  to  the  OIG  when  it  is 
completed  and  approved. 

The  report  is  critical  of  the  STARS  Program's  provisions  for 
participation  by  the  Services.  It  does  not  acknowledge  the  fact 
that  the  Services  initially  had  representatives  in  the  Program 
Office  and  unilaterally  chose  to  remove  them.  The  opportunity 
for  Service  participation  in  planning  and  managing  the  program 
has  always  been  there.  In  addition,  it  should  be  obvious  that 
reduced  funding  would  necessarily  reduce  the  ability  of  the 


APPENDIX  B 
Page  2  of  6 


20 


•program  to  fund  Service's  proposed  efforts,  and  that  the 
available  resources  would  be  focused  on  the  most  central  parts  of 
the  program,  namely  the  Competing  Primes.  Nevertheless,  DARPA 
has  already  taken  the  initiative  to  work  more  closely  with  the 
Services  to  expand  their  involvement. 

Although  we  commonly  refer  to  the  software  development 
process,  it  is  not  a  singular  well-defined  set  of  methods  and 
activities,  and  is  not  yet  a  true  engineering  discipline. 

However,  our  dependence  on  software  is  such  that  we  must  act  to 
improve  the  process  whether  it  embodies  a  mature  engineering 
discipline  or  not.  Under  these  circumstances,  we  should  expect 
that  a  research  and  development  program  like  STARS,  whose  main 
objective  is  to  develop  means  to  improve  this  process,  will  be 
refined  and  adjusted  as  our  experience  with  the  process  and  its 
evolving  methods,  techniques  and  tools  grows. 

The  comment  in  the  report  that  "The  main  objective  of  an 
improved  software  engineering  process  is  to  allow  software  to  be 
designed  before  hardware  is  selected,"  totally  misses  the  mark  on 
what  a  software  engineering  process  is,  and  why  the  improvements 
are  needed.  Making  a  selection  decision  on  the  computer  hardware 
for  a  software  intensive  system  prior  to  designing  that  software 
is  usually  a  reflection  of  poor  management  judgement,  and  more 
often  than  not  leads  to  disaster  later  in  the  program.  Although 
correcting  such  management  practice  is  very  important,  it  is 
unrelated  to  the  objective  of  improving  the  software  engineering 
practice.  The  true  objective  is  to  achieve  higher  quality,  more 
reliable,  more  maintainable  software,  in  a  more  cost  effective 
and  consistent  manner. 

RESPONSE  TO  DRAFT  AUDIT  REPORT  RECOMMENDATIONS 

The  report  recommendations  are  repeated  below,  followed  by 
the  OUSD(A)  and  DARPA  responses  and  comments: 

1.  We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition,  in  consonance  with  the  developing  DoD  Software 
Master  Plan: 

[Comment:  The  DoD  Software  Master  Plan  Is  currently  In  coordination.  It  is  inappropriate  to  cite  such 
a  Plan  in  the  Audit  Report  before  it  Is  finalized  and  approved.] 


a.  Direct,  and  provide  appropriate  support  for,  the  Defense 
Advanced  Research  Projects  Agency  to: 

(1)  identify,  in  coordination  with  the  Services, 
adequate  resources  for  the  Software  Technology  for  Adaptable, 
Reliable  Systems  Program,  and 
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*  *  RESPONSE:  Partially  concur.  DARPA  has  already  identified  additional  resources  in  the  current 
Budget  Estimate  Submission.  This  level  of  funding,  when  combined  with  industry  investment  in  STARS 
compatible  technology  by  the  STARS  Primes,  is  adequate  to  build  a  viable  STARS  capability.  While 
some  additional  funding  would  be  beneficial,  it  cannot  be  accommodated  under  the  current  austere  and 
uncertain  budget  environment.  The  role  of  the  Services  is  primarily  one  of  transitioning  and 
exploiting  the  products  of  the  STARS  Program.  The  STARS  Program  Manager  is  already  working  closely 
with  the  Services  to  define  how  their  role  should  be  executed  in  the  current  program.  Any  required 
Service  funding  will  be  identified  when  the  Services*  efforts  are  fully  defined. 

Corrective  action  has  been  initiated.  Completion  is  expected  by  the  end  of  FY 1991. 


(2)  develop  concrete  financial  plans  to  complete  the 
Software  Technology  for  Adaptable,  Reliable  Systems  Program  in  a 
timely  manner. 

RESPONSE:  Concur,  STARS  has  suffered  seriously  from  organizational  problems  and  significant  loss  of 
funding  in  the  past.  DARPA  transferred  the  STARS  Program  to  the  Information  Science  and  Technology 
Office  in  November  1989,  in  order  to  ensure  appropriate  planning,  technical  oversight,  and  budget 
review.  Since  that  time  DARPA  has  included  increased  funding  in  its  Budget  Estimate  Submission.  A 
next  level  detailed  execution  plan  with  an  associated  review  of  resource  reasonableness  is  in 
progress.  This  plan  will  provide  more  detailed  traceability  between  funding  elements  and 
incremental  products. 

Corrective  action  has  been  initiated.  Completion  expected  by  end  of  FY  1991. 

b.  Direct  each  of  the  Services  to  assign  a  representative 
to  work  directly  and  be  collocated  with  the  Software  Technology 
for  Adaptable t  Reliable  Systems  Program's  Joint  Program  Manager. 

RESPONSE:  Partially  concur.  While  the  Services  should  designate  a  lead  liaison  officer  to  work  with 
the  STARS  Program  Manager,  in  this  decreasing  budget  and  manpower  era,  other  solutions  need  to  be 
found.  As  stated  above  the  Services’  current  role  is  to  prepare  for  the  transition  and  exploitation 
of  STARS  products.  When  the  Services*  effort  increases  to  require  full-time  liaison  with  the  STARS 
Program,  DARPA  should  negotiate  with  the  Services  to  have  their  representatives  be  collocated  with 
the  STARS  Program  Office, 

Corrective  action  has  been  initiated.  Completion  is  expected  by  the  end  of  FY  1991. 


c.  Evaluate  the  progress  of  the  Software  Technology  for 
Adaptable,  Reliable  Systems  annually  to  determine  whether  to 
continue  the  Program. 

RESPONSE:  Partially  concur.  We  fully  concur  with  the  intent  of  this  recommendation  but  think  that 
the  Program  is  currently  being  adequately  reviewed  and  managed  within  DARPA  and  its  Information 
Sciences  and  Technology  Office.  The  changes  in  management  structure  and  personnel  were  made  to 
accomplish  the  management  improvements  now  in  place.  DARPA  will  thoroughly  review  the  STARS  program 
annually  to  determine  whether  continuation  is  appropriate.  No  new  action  is  planned. 
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d.  Consolidate  the  management  of  technical  activities  of 
*  the  Ada  Joint  Program  Office,  the  Software  Engineering  Institute, 
and  the  Software  Technology  for  Adaptable,  Reliable  Systems  Joint 
Program  Office. 

RESPONSE:  Do  not  concur.  The  management  of  the  Software  Engineering  Institute  (SEI)  and  the  STARS 
Program  have  been  consolidated  for  several  years.  The  Ada  Joint  Program  Office’s  primary  activity  is 
updating  the  Ada  Standard  in  accordance  with  the  requirements  or  the  American  National  Standards 
Institute  (ANSI)  and  the  International  Standards  Organization  (ISO).  The  technical  efforts  in  the 
program  are  actually  being  managed  within  the  Services,  and  are  not  closely  related  to  either  STARS 
or  the  SEI  programs.  There  is  already  adequate  technical  coordination  between  the  two  programs 
through  several  mechanisms. 


2.  We  recommend  that  the  Director,  Defense  Advanced  Research 
Projects  Agency: 


a.  Require  that  the  Software  Technology  for  Adaptable, 
Reliable  Systems  Program  to: 

(1)  establish  quantitative  program  goals, 

(2)  document  a  performance  baseline,  and 

(3)  select  and  implement  a  system  for  measuring 
progress  in  goal  achievement. 

RESPONSE:  Partially  concur.  Quantitative  goals  would  be  artificial,  and  therefore,  not  meaningful 
for  STARS.  However,  The  STARS  Program  Director  is  conducting  a  consolidated  planning  effort  to 
establish  a  performance  baseline,  clear  goals  for  each  technical  area,  and  an  evaluation  framework 
for  measuring  progress  in  their  achievement.  The  SEI  has  been  tasked  to  lead  a  team  to  develop  the 
framework.  They  will  evaluate  the  success  of  STARS  technology  in  application  demonstrations  to  be 
conducted  in  the  FY94-95  timeframe. 

Corrective  action  has  been  initiated.  Completion  is  expected  by  the  end  of  FY 1991. 


b.  Require  that  the  Joint  Program  Manager  develop, 
document,  and  maintain  an  effective  system  of  management  control 
objectives,  requirements,  and  procedures  unique  to  the  Software 
Technology  for  Adaptable,  Reliable  Systems  Program  and 
incorporate  them  into  the  revised  Program  Management  Plan. 

RESPONSE:  Concur.  DARPA  is  revising  the  STARS  Program  Management  Plan  (PMP),  and  has  already 
produced  the  first  draft.  The  PMP  will  reflect  the  oganizational,  financial,  and  program  execution 
changes  since  1986.  The  Plan  will  also  incorporate  the  appropriate  information  to  support 
fulfillment  of  this  recommendation. 

Corrective  action  has  been  initiated.  Completion  is  expected  by  the  end  of  FY  1991. 
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c.  Designate  the  Software  Technology  for  Adaptable, 
Reliable  systems  Program  as  a  major  program  and  manage  it 
accordingly. 

RESPONSE:  Concur.  DARPA  will  designate  STARS  as  a  major  program  to  be  managed  in  accordance  with 
DARPA  Instruction  No.  13.  The  revised  PMP  will  serve  as  the  detailed  program  plan  required  by 
Instruction  No.  13.  DARPA  will  ensure  that  the  STARS  Program  receives  appropriate  review,  botii 
internally  and  externally,  to  provide  necessary  monitoring  and  evaluation  of  program  progress,  status 
and  potential  risks. 

Corrective  action  has  been  initiated.  Completion  is  expected  by  the  end  of  FY 1991. 


d.  Develop  and  implement  procedures  to  consistently 
identify  major  programs  as  defined  in  Defense  Advanced  Research 
Projects  Agency  Instruction  No.  13. 

RESPONSE:  Concur.  DARPA  was  already  examining  its  structure,  management  systems,  and  procedures 
with  respect  to  DoD’s  Total  Quality  Management  (TQM)  philosophy.  Instruction  13  will  be  reviewed  and 
updated  as  part  of  DARPA’s  current  TQM  effort  to  ensure  procedures  are  in  place  to  consistently 
identify  major  programs. 

Corrective  action  has  been  initiated.  Completion  Is  expected  by  the  end  of  FY  1991. 
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DEFENSE  ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILSON  BOULEVARD 
ARLINGTON,  VA  22209-2308 


NOV  1 3  1990 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  Report  on  the  inspection  of  the  Software  Technology  for  Adaptable, 
Reliable  Systems  (STARS)  Program  (Project  No.  OFE-0007) 


In  response  to  your  memorandum  of  August  31,  1990,  enclosed  are  comments  to 
the  draft  report  recommendations.  Only  those  recommendations  addressed  to  the 
Defense  Advanced  Research  Projects  Agency  have  been  included. 

While  the  response  format  deals  only  with  the  specific  recommendations,  an 
effort  has  been  made  to  integrate  our  understanding  of  the  report’s  issues  and  general 
discussion  sections. 

In  all  areas,  remedies  are  already  underway.  Since  the  audit,  the  STARS 
Program  has  made  progress  in  addressing  the  acknowledged  deficiencies.  Any  further 
discussions  which  might  be  of  benefit  to  ongoing  and  future  changes  are  welcomed  and 
encouraged.  The  point  of  contact  is  Dr.  Jack  Kramer  at  (703)  243-8655. 

Victor  H.  Reis 
Acting  Director 


Attachment: 

Response 
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Responses  to  Recommendations 

of  the  Inspector  General,  Department  of  Defense,  Draft  Report  on  the 
Audit  of  Management  of  the  Software  Technology  for  Adaptable,  Reliable  Systems 
(STARS)  Program  (Project  No.  OFE-0007)  Conducted  October  1989  through  June  1990 


Recommendation  1:  Addressed  to  the  Under  Secretary  of  Defense  (Acquisition). 

Recommendation  2a:  The  Director,  Defense  Advanced  Research  Projects  Agency  require 
the  Software  Technology  for  Adaptable,  Reliable  Systems  Program  to: 

(1)  establish  quantitative  program  goals, 

(2)  document  a  performance  baseline,  and 

(3)  select  and  implement  a  system  for  measuring  progress  in  goal  achievement. 

Response:  Partially  concur.  Quantitative  goals  would  be  artificial,  and  therefore,  not 
meaningful  for  STARS.  However,  the  STARS  Program  Director  is  conducting  a 
consolidated  planning  effort  to  establish  a  performance  baseline,  clear  goals  for  each 
technical  area,  and  an  evaluation  framework  for  measuring  progress  in  their  achievement. 
The  SEI  has  been  tasked  to  lead  a  team  to  develop  the  framework.  They  will  evaluate 
the  success  of  STARS  technology  in  application  demonstrations  to  be  conducted  in 
FY94-95  timeframe. 

Corrective  action  has  been  initiated.  Completion  expected  during  FY  1991. 

Recommendation  2b:  The  Director,  Defense  Advanced  Research  Projects  Agency,  requires 
that  the  Joint  Program  Manager  develop,  document,  and  maintain  an  effective  system  of 
management  control  objectives,  requirements,  and  procedures  unique  to  the  Software 
Technology  for  Adaptable,  Reliable  Systems  Program  and  incorporate  them  into  the 
revised  Program  Management  Plan. 

Response:  Concur.  DARPA  is  revising  the  STARS  Program  Management  Plan  (PMP), 
and  has  already  produced  the  first  draft.  The  PMP  will  reflect  the  organizational, 
financial,  and  program  execution  changes  since  1986.  The  Plan  will  also  incorporate  the 
appropriate  information  to  support  fulfillment  of  this  recommendation. 

Corrective  action  has  been  initiated.  Completion  expected  during  FY  1991. 
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Recommendation  2c:  The  Director,  Defense  Advanced  Research  Projects  Agency, 
designate  the  Software  Technology  for  Adaptable  Reliable  Systems  Program  as  a  major 
program  and  manage  it  accordingly. 

Response:  Concur.  DARPA  will  designate  STARS  as  a  major  program  to  be  managed 
in  accordance  with  DARPA  Instruction  No.  13.  The  revised  PMP  will  serve  as  the 
detailed  program  plan  required  by  Instruction  No.  13.  DARPA  will  ensure  that  the 
STARS  Program  receives  appropriate  review,  both  internally  and  externally,  to  provide 
necessary  monitoring  and  evaluation  of  program  progress,  status  and  potential  risks. 
Corrective  action  has  been  initiated.  Completion  is  expected  during  FY  1991. 

Recommendation  2d:  The  Director,  Defense  Advanced  Research  Projects  Agency,  develop 
and  implement  procedures  to  consistently  identify  major  programs  as  defined  in  Defense 
Advanced  Research  Projects  Agency  Instruction  No.  13. 

Response:  Concur.  DARPA  was  already  examining  its  structure,  management  systems, 
and  procedures  with  respect  to  DoD’s  Total  Quality  Management  (TQM)  philosophy. 
Instruction  13  will  be  reviewed  and  updated  as  part  of  DARPA’s  current  TQM  effort  to 
ensure  procedures  are  in  place  to  consistently  identify  major  programs. 

Corrective  action  has  been  initiated.  Completion  expected  during  FY  1991. 
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OFFICE  OF  THE  COMPTROLLER  OF  THE  DEPARTMENT  OF  DEFENSE 
WASHINGTON,  DC  20301-1100 


NOV  5  1990 


(Information  Resources 
Management) 


MEMORANDUM  FOR  DEPUTY  ASSISTANT  INSPECTOR  GENERAL  FOR  INSPECTION 

GAO  AND  AUDIT  FOLLOWUP 

SUBJECT:  OIG  Report,  "Draft  Report  on  the  Audit  of  Management 

of  the  Software  Technology  for  Adaptable,  Reliable 
Systems  (STARS)  Program  (Project  No.  OFE-0007)." 


In  your  August  31,  1990  memorandum,  you  requested  we 
provide  comments  on  the  "Draft  Report  on  the  Audit  of  Management 
of  the  Software  Technology  for  Adaptable,  Reliable  Systems 
(STARS)  Program  (Project  No.  OFE-0007)."  The  subject  report  has 
been  reviewed  and  we  concur  with  the  reports  recommendations. 
Additionally,  before  funding  is  approved,  the  Director  of  DARPA 
should  be  required  to  ensure  that  STARS  research  projects  do  not 
duplicate  other  research  projects. 

The  Office  of  the  Deputy  Comptroller  action  officer  is 
Mr.  Burt  Newlin  at  695-2554. 


Cynthia  Kendall 

Deputy  Comptroller  (Information 
Resources  Management) 
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SUMMARY  OF  POTENTIAL  MONETARY  AND 
OTHER  BENEFITS  RESULTING  FROM  AUDIT 


Recommendation 

Reference 


Category  and  Description  of  Benefit 


Amount  and/or 
Type  of  Benefit 


1  .  a  • 


Program.  Products  of  STARS  made  Undeterminable  monetary 

available  in  a  timely  manner.  benefit.  Savings  cannot 

be  determined  until  results 
of  STARS  are  actually  used. 


l.b. 


1  .c . 


l.d. 


2. a. 


2.b. 


2.c. 


2.d. 


Economy  and  Efficiency.  Enhanced 
Service  involvement,  better 
coordination  with  related  software 
research  by  Services. 

Economy  and  Efficiency.  DoD 
decision  on  best  use  of  funds 
budgeted  for  STARS. 

Economy  and  Efficiency. 

Centralized  management  of  related 
DoD  Software  Initiative 
technical  projects. 

Program.  Clear  indicators 
of  STARS  progress. 


Internal  Control.  More 
effective  managerial  control 
over  STARS  resources. 


Compliance  with  DARPA 
Instruction  No.  13. 


Internal  Control.  Provide  for 
the  uniform  application  of 
DARPA  Instruction  No.  13. 


Nonmonetary.  More 
efficient  and  effective 
management  of  STARS. 


Undeterminable  monetary 
benefit.  Savings  are 
unknown  unless  Program  is 
completed  or  terminated. 

Nonmonetary.  More 
effective  use  of  DoD 
software  research 
resources . 

Nonmonetary .  Improved 
ability  to  judge  STARS* 
effectiveness  and  likely 
economic  worth. 

Nonmonetary.  Establish¬ 
ment  and  documentation  of 
STARS  management  control 
objectives  and  procedures. 

Nonmonetary.  More 
stringent  DARPA  oversight 
and  detailed  planning  by 
STARS  program  managers. 

Nonmonetary.  Consistent 
management  of  DARPA  programs. 
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ACTIVITIES  VISITED  OR  CONTACTED 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Washington,  DC 
Comptroller  of  the  Department  of  Defense,  Washington,  DC 
Director  of  Defense  Research  and  Engineering,  Washington,  DC 

Department  of  the  Army 

Director  of  Information  Systems  for  Command, 

Control,  Communications  and  Computers,  Washington,  DC 

Department  of  the  Navy 

Space  and  Naval  Warfare  Systems  Command,  Washington,  DC 
Department  of  the  Air  Force 

Air  Force  Systems  Command,  Andrews  Air  Force  Base,  MD 
Electronic  Systems  Division,  Hanscom  Air  Force  Base,  MA 

Defense  Agencies 

Defense  Advanced  Research  Projects  Agency,  Arlington,  VA 
Defense  Contract  Audit  Agency,  Alexandria,  VA 

Non-DoD  Activities 


National  Aeronautics  and  Space  Administration,  Greenbelt,  MD 
Non-Government  Activities 


Boeing  Aerospace  and  Electronics,  Kent,  WA 
International  Business  Machines,  Gaithersburg,  MD 
Software  Engineering  Institute,  Pittsburgh,  PA 
Unisys,  Reston,  VA 
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AUDIT  TEAM  MEMBERS 


Nancy  L.  Butler,  Director  for  Financial  Management  Programs 

Terry  L.  McKinney,  Program  Director 

James  W.  Hutchinson,  Project  Manager 

James  F.  Friel,  Auditor 

Frederick  G.  Sacchet,  Auditor 

Susanne  B.  Allen,  Editor 
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FINAL  REPORT  DISTRIBUTION 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition 
Comptroller  of  the  Department  of  Defense 

Department  of  the  Army 

Secretary  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management) 
Department  of  the  Navy 
Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 
Department  of  the  Air  Force 
Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management 
and  Comptroller) 

Defense  Agencies 

Director,  Defense  Advanced  Research  Projects  Agency 
Non-DoD  Activities 


Office  of  Management  and  Budget 
U.S.  General  Accounting  Office, 

NSIAD  Technical  Information  Center 

Congressional  Committees: 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 

Senate  Ranking  Minority  Member,  Committee  on  Armed  Services 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Ranking  Minority  Member,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Subcommittee  on  Government  Operations 
House  Subcommittee  on  Legislation  and  National  Security, 
Committee  on  Government  Operations 
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INTERNET  DOCUMENT  INFORMATION  FORM 


A .  Report  Title:  Management  of  the  Software  Technology  for  Adaptable, 
Reliable  Systems  Program 

B.  DATE  Report  Downloaded  From  the  Internet:  08/01/00 


C.  Report’s  Point  of  Contact:  (Name,  Organization,  Address,  Office 
Symbol,  &  Ph  #):  OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General,  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-2884 


D.  Currently  Applicable  Classification  Level:  Unclassified 

E.  Distribution  Statement  A:  Approved  for  Public  Release 

F.  The  foregoing  information  was  compiled  and  provided  by: 

DTIC-OCA,  Initials: _ VM _ Preparation  Date  08/01/00 


The  foregoing  information  should  exactly  correspond  to  the  Title,  Report  Number,  and  the  Date 
the  accompanying  report  document.  If  there  are  mismatches,  or  other  questions,  contact  the 
above  OCA  Representative  for  resolution. 


